Skip to content

Revert "Build PyTorch wheel via PEP 517 instead of setup.py" (#21562) - #21684

Closed
shoumikhin wants to merge 1 commit into
mainfrom
revert-21562-riscv-break
Closed

Revert "Build PyTorch wheel via PEP 517 instead of setup.py" (#21562)#21684
shoumikhin wants to merge 1 commit into
mainfrom
revert-21562-riscv-break

Conversation

@shoumikhin

Copy link
Copy Markdown
Contributor

Reverts the install_pytorch.sh change from #21562.

Why

#21562 broke test-riscv on main. 17 of 17 jobs, on every commit that has landed since it went in. Failing step is pip install --no-build-isolation extension/llm/tokenizers:

[299/312] Generating CXX dyndep file CMakeFiles/pytorch_tokenizers_cpp.dir/CXX.dd
FAILED: CMakeFiles/pytorch_tokenizers_cpp.dir/src/python_bindings.cpp.o
cc1plus: error: to generate dependencies you must specify either '-M' or '-MM'

Cause

The PEP 517 change itself is correct and should come back. The problem is how the build requirements are installed.

To run python -m build --wheel --no-isolation, #21562 installs build, scikit-build-core, setuptools, cmake and ninja permanently into the image's conda environment. scikit-build-core ships a setuptools plugin that registers itself as the build_ext command class. Every later pip install --no-build-isolation in that image then runs its CMake configure instead of the plain setuptools one, which switches on C++20 module dependency scanning. The image's GCC 14.2.0 cannot produce the scan output CMake asks for, so the tokenizers build dies.

Two greps confirm it. dyndep has zero hits in the last green job log and one in the red one. scikit_build_core is absent from the green log and present in the red traceback.

Why CI did not catch it

test-riscv lives in its own workflow, Test RISC-V Backend. It does not run on pull requests unless the change touches specific RISC-V paths, it is not a required merge context, and it does not gate viable/strict. So nothing in the pre-merge gate could have seen this.

Next

Re-landing separately with the build requirements confined to a throwaway virtualenv, so they stop leaking into every later build in the image. That PR will carry a ciflow/trunk tag so test-riscv actually runs on it before it merges.

This reverts the install_pytorch.sh change from #21562. It broke test-riscv on main, 17 of 17 jobs, on every commit since it landed.

The PEP 517 change itself is correct. The problem is that the PR installs the PEP 517 build requirements permanently into the image conda environment. scikit-build-core registers a setuptools build_ext plugin, so every later "pip install --no-build-isolation" in that image runs its cmake configure instead of the plain setuptools one. That turns on C++20 module dependency scanning, which the image GCC 14.2.0 cannot satisfy, and the tokenizers build fails with "cc1plus: error: to generate dependencies you must specify either -M or -MM".

Reverting to unblock main. Re-landing separately with the build requirements confined to a throwaway virtualenv.
Copilot AI lite review requested due to automatic review settings August 8, 2026 00:05
@pytorch-bot

pytorch-bot Bot commented Aug 8, 2026

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21684

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure

As of commit a02e38d with merge base a439afd (image):

NEW FAILURE - The following job has failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot Bot added the ci-no-td label Aug 8, 2026
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Aug 8, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@shoumikhin shoumikhin added the release notes: none Do not include this in the release notes label Aug 8, 2026
shoumikhin added a commit that referenced this pull request Aug 8, 2026
Forward fix for #21562.

#21562 moved the PyTorch wheel build from `setup.py bdist_wheel` to
`python -m build --wheel --no-isolation`. That direction is correct and
is kept here. The problem was that it installed the PEP 517 build
requirements permanently into the image's conda environment, and two
different things in that install changed how every later build in the
image behaves. Both broke main.

### 1. All 17 RISC-V jobs failed

scikit-build-core ships a setuptools plugin that registers itself as the
`build_ext` command class. Once it is present in the environment, any
later `pip install --no-build-isolation` picks it up. The tokenizers
build is exactly that, so its CMake configure started running through
scikit-build-core, which turns on C++20 module dependency scanning. The
image compiler cannot produce the scan output:

```
[299/312] Generating CXX dyndep file CMakeFiles/pytorch_tokenizers_cpp.dir/CXX.dd
FAILED: CMakeFiles/pytorch_tokenizers_cpp.dir/src/python_bindings.cpp.o
cc1plus: error: to generate dependencies you must specify either '-M' or '-MM'
```

### 2. The wheel was built without LAPACK

The same install also put a pip `cmake` wheel into the environment,
which lands on PATH ahead of the image's own cmake. The two are the same
version but not interchangeable: the pip wheel does not search the conda
environment, so `FindMKL` stopped finding the MKL libraries that are
installed there and PyTorch was configured with MKL and LAPACK off. 12
tests in `extension/llm/modules/test/test_turboquant_kv_cache.py` then
failed with:

```
RuntimeError: Calling torch.geqrf on a CPU tensor requires compiling PyTorch with LAPACK.
```

### Fix

Install the build requirements into a throwaway venv instead of into the
image, and invoke the build through that venv's interpreter by absolute
path. The venv is created with `--system-site-packages`, so PyTorch
still builds against the image's numpy and toolchain and still reuses
sccache. The venv is deleted once the wheel is built, so neither the
scikit-build-core plugin nor the pip cmake survives into the image, and
the image's own cmake stays in front.

### Evidence

From the docker image build log, PyTorch's configure summary:

| | #21562, as merged | this PR |
| --- | --- | --- |
| cmake actually used | `.../site-packages/cmake/data/bin/cmake` |
`/opt/conda/envs/py_3.10/bin/cmake` |
| `USE_MKL` | `OFF` | `ON` |
| `USE_LAPACK` | `0` | `1` |

CI on this PR:

- `test-riscv`: 17 of 17 green. On main it is 17 of 17 red.
- `test-lora-multimethod-linux`: green. On main it is red.

### Note on the jobs that did not finish

`unittest / linux`, `unittest-editable / linux` and `test-lora-linux`
were all cancelled at 1 hour 31 minutes, which is the 90 minute job cap.
Any PR that touches `.ci/docker/` forces a full image rebuild that
consumes roughly 30 minutes of that budget, so these three cannot
complete on a PR of this shape. That is also why the LAPACK problem was
not visible on #21562 before it merged. The `USE_LAPACK` line in the
image build log above is the direct evidence for that half of the fix,
since it is read at configure time rather than at test time.

### Relationship to the revert

This branch contains two commits, a revert of #21562 followed by the
corrected version. The net diff against main is a single file,
`.ci/docker/common/install_pytorch.sh`, and squash merge collapses them
into one commit.

This supersedes #21684, which is a straight revert of #21562. If this
lands, #21684 should be closed without merging. Do not merge #21684
after this one, since it would restore the previous file wholesale and
undo this fix.
@shoumikhin

Copy link
Copy Markdown
Contributor Author

closing this without merging. #21685 landed and fixed the root cause instead of reverting: it builds the wheel in a venv so the pip cmake/ninja and scikit-build-core installed into the ci docker image stop shadowing the image toolchain for later builds.

all 7 red groups from #21562 are green on main now, including test-riscv 17/17 and deit_e2e back to about 22 minutes. merging this revert on top would undo that fix.

@shoumikhin shoumikhin closed this Aug 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-no-td CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. release notes: none Do not include this in the release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants